Skip to content

Reconcile Role resources with a non-superuser admin such as the master user of managed PostgreSQL cloud services - #71

Open
ThoSap wants to merge 9 commits into
mainfrom
fix-role-reconcile-without-pg-authid-access
Open

Reconcile Role resources with a non-superuser admin such as the master user of managed PostgreSQL cloud services#71
ThoSap wants to merge 9 commits into
mainfrom
fix-role-reconcile-without-pg-authid-access

Conversation

@ThoSap

@ThoSap ThoSap commented Sep 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Fixes #70

The Role controller failed on AWS RDS with permission denied for table pg_authid.
The same happens on every managed PostgreSQL service and on any vanilla PostgreSQL where the admin/management user is usually not a superuser, not do the cloud offerings support crating a superuser.
This PR makes the Role controller work with an admin role that has only LOGIN, CREATEDB, and CREATEROLE.

What changed

  1. Reads use pg_roles instead of pg_authid. Role state and membership come from the public view pg_roles and the public catalog pg_auth_members. pg_authid stays in the generated jOOQ sources for the test helper only.
  2. ALTER ROLE names only the options that differ from the current state. PostgreSQL rejects NOSUPERUSER, NOREPLICATION, and NOBYPASSRLS from a non-superuser even when the value does not change, so the previous full statement failed on every update of an existing role.
  3. The password hash comparison is replaced by a keyed fingerprint. The hash in pg_authid cannot be read without superuser rights. The operator now stores an HMAC-SHA256 of the Secret password in status.passwordFingerprint and compares against it on each reconcile. The key is random and lives in a Secret in the operator namespace (postgresql-operator.password-fingerprint.secret-name). A reader of the Role status learns nothing about the password without that key. The RBAC rule for secrets gains create for this one Secret.
  4. The operator sends a SCRAM-SHA-256 verifier instead of the cleartext password. The cleartext never reaches the server or its statement log. A Secret that already holds a verifier passes through unchanged. The new spec field passwordEncryption: server opts out for MD5-only clients. CloudNativePG does the same since v1.29.2.
  5. New test RoleReconcilerNonSuperuserTest runs the controller with a LOGIN CREATEDB CREATEROLE admin. It covers create, update, password rotation, login toggle, membership, drop, and the error path for superuser: true.

Documented consequences

  • Minimum admin privileges per Custom Resource, see docs/cluster-connection.md.
  • A non-superuser admin cannot set superuser, replication, or bypassrls.
  • On PostgreSQL 16+, the admin can only alter roles on which it holds ADMIN OPTION.
  • A password change made directly in PostgreSQL is not detected. The Secret is the source of truth.
  • Every existing Role gets one password update after the upgrade, because its status has no fingerprint yet.
  • A pre-hashed password bypasses server-side password policies such as credcheck or the Cloud SQL password policy.

Verification against a non-superuser admin

Run on PostgreSQL 15 and 18 as LOGIN NOSUPERUSER CREATEDB CREATEROLE, which mirrors the master user of the managed services.

Check PG 15 PG 18
SELECT on pg_authid denied, 42501 denied, 42501
SELECT on pg_shadow denied, 42501 denied, 42501
SELECT on pg_roles, all flag columns ok, password masked ok, password masked
SELECT on pg_auth_members ok ok
shobj_description(oid, 'pg_authid') with the oid from pg_roles ok ok
Previous buildAlterRole statement denied, 42501 denied, 42501
ALTER ROLE ... NOSUPERUSER / NOREPLICATION / NOBYPASSRLS alone denied denied
ALTER ROLE with login, password, createdb, createrole, inherit, connection limit, valid until ok ok
GRANT / REVOKE membership on a role the admin created ok ok
ALTER ROLE ... PASSWORD '<SCRAM verifier>' stored verbatim, login works stored verbatim, login works

Managed PostgreSQL offerings

Offering Admin role pg_authid
AWS RDS for PostgreSQL, Amazon Aurora rds_superuser denied
Google Cloud SQL, AlloyDB cloudsqlsuperuser, alloydbsuperuser denied by default, flag cloudsql.pg_authid_select_role can grant it
Azure Database for PostgreSQL Flexible Server azure_pg_admin denied
Neon neon_superuser denied for all but the project owner
Supabase postgres denied since the 2023 demotion
Aiven, DigitalOcean, Heroku, Timescale Cloud, Scaleway, OVH, IBM Cloud provider admin role denied
Crunchy Bridge, self-hosted, CloudNativePG, Zalando, Crunchy PGO, StackGres, Percona postgres readable, real superuser

The change works on all of them, because it uses only public catalogs and never names superuser-only options without a change.

Tests

@ThoSap ThoSap self-assigned this Sep 11, 2026
@ThoSap ThoSap added bug Something isn't working documentation Improvements or additions to documentation enhancement New feature or request labels Sep 11, 2026
@ThoSap
ThoSap requested a review from stplasim September 11, 2026 10:46
@ThoSap
keep the cause of a failed password fingerprint key load in the `Role…
fe0ad29 
…` status

The catch block around the HMAC computation also caught every error from the load of the key Secret, such as a 403 from the Kubernetes API, and reported it as "HmacSHA256 not available". The key is now loaded before the try block, and the catch covers only the two checked exceptions of the HMAC API.

The new test covers the creation of the key Secret with a 32 byte key, the reuse of an existing key by a second operator process, a new key after the Secret is lost, the error for a Secret without the `key` entry, and the exact HMAC construction that existing `Role` statuses depend on.
@ThoSap
record the password fingerprint in the Role status only after the d…
15cdd8f 
…atabase commit

The fingerprint was written to the status inside the transaction. When the commit failed, the error handler still patched the status with the new fingerprint, and the next reconcile saw no password change. The fingerprint and the server password are now computed before the transaction, and the status receives the fingerprint after the transaction returns.

The new test covers the state of every `Role` after the upgrade to the version that introduced the fingerprint. It changes the password in PostgreSQL, removes the fingerprint from the status, and asserts that the Secret password is applied once and left alone afterwards.
@ThoSap
grant create on Secrets through a namespaced Role in the operator n…
312dc94 
…amespace

The `create` verb was part of the ClusterRole of the `Role` controller, which allowed the operator to create Secrets in every namespace. It is only needed for the password fingerprint key Secret in the operator namespace. The Quarkus Kubernetes extension now generates a namespaced Role and RoleBinding for it. The default RoleBinding to the `view` ClusterRole is kept explicitly, because configured role bindings replace it.
@ThoSap
test superuser-only flags and password modes with a non-superuser admin
70ec3a9 
The flag test now covers `superuser`, `replication`, and `bypassrls`. Two tests run `passwordEncryption: server` and a pre-hashed SCRAM-SHA-256 verifier under the non-superuser admin.
@ThoSap ThoSap mentioned this pull request Sep 11, 2026
Closed
@ThoSap
clear the password of a NOLOGIN role that still holds one
3163512 
`buildAlterRole` added `PASSWORD NULL` only on the transition from `LOGIN` to
`NOLOGIN`. A role that was already `NOLOGIN` but still held a password kept it.

`pg_roles` masks `rolpassword` with a constant, so the operator cannot tell
whether such a role has a password. It therefore always clears the password when
the spec expects no login.

This costs nothing in the steady state. `RoleReconciler` calls `alterRole` only
when the login state, the flags, or the password differ from the spec.
@ThoSap
document the threat model of the Role password fingerprint
65a77d1 
The fingerprint is a keyed hash, not a password hash with a work factor. Without
the key it reveals nothing about the password. With the key an attacker can test
guesses offline at `HMAC-SHA256` speed, so the key Secret is a credential.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@stplasim stplasim Awaiting requested review from stplasim

At least 1 approving review is required to merge this pull request.

Assignees

@ThoSap ThoSap

Labels

bug Something isn't working documentation Improvements or additions to documentation enhancement New feature or request

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Rermission denied for table pg_authid (SQLSTATE 42501) on AWS RDS

1 participant

@ThoSap